Skip to content

Add Anthropic Message Batches support - #6713

Open
ricken07 wants to merge 7 commits into
spring-projects:mainfrom
ricken07:anthropic-batch-support
Open

Add Anthropic Message Batches support#6713
ricken07 wants to merge 7 commits into
spring-projects:mainfrom
ricken07:anthropic-batch-support

Conversation

@ricken07

Copy link
Copy Markdown
Contributor

The spring-ai-anthropic module exposes AnthropicChatModel for synchronous and streaming calls, but there is no abstraction for Anthropic's Message Batches API. Applications that need bulk processing at reduced cost have to create a second Anthropic SDK client themselves and pin their own SDK version, which drifts from Spring AI on HTTP configuration, headers and observability.

This adds AnthropicBatchModel alongside the chat model, covering the five provider operations: submit, retrieve, results, cancel and delete.

ricken07 added 7 commits July 31, 2026 12:21
Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Add AnthropicBatchModel, exposing the five provider operations of the
Message Batches API, and its default implementation over the official
Anthropic Java SDK.

Batch processing is asynchronous and can take up to 24 hours, so this is
deliberately not a ChatModel: submit has no immediate ChatResponse to
return. The control operations are synchronous; only results is
reactive, backed by Flux.using so the SDK stream is closed on
termination and on cancellation and a batch with a very large number of
entries is never held in memory.

Entries are mapped through the same Prompt conversion as
AnthropicChatModel.call, so prompt caching, thinking, images and PDF
documents, structured output and tool definitions behave identically on
both paths, and a succeeded result is converted back into a
ChatResponse with the same generations, metadata and usage. Tool calls
are not executed: a batch entry cannot be continued mid-flight, so a
tool_use response is returned as is.

Spring AI provides the provider access, not the orchestration. There is
no polling, no persistence and no scheduling: those stay with the
application, which owns the batch id and the customId correlations.

Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Expose the batch model as a Spring Boot bean so an application can
inject it without building an SDK client itself.

The bean is opt-in behind spring.ai.anthropic.batch.enabled, so
applications that never submit batches do not pay for a second HTTP
client. Connection settings and the chat model defaults are reused
as-is, which means a batch shares the credentials, base URL, timeout,
retries, proxy, custom headers and HTTP client customizers of realtime
calls; spring.ai.anthropic.batch.model and .max-tokens override the
model and output ceiling for batch entries only.

Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Cover the submit, poll, read, cancel and error paths against the real
API, which the mocked unit tests cannot validate: the JSONL result
stream, the out-of-order correlation by customId and the usage reported
per entry.

Two switches guard it, because a batch consumes quota and its
completion time is not bounded by the API contract: ANTHROPIC_API_KEY
must be set, the repository-wide convention, and
ANTHROPIC_BATCH_IT_DISABLED turns it off even when a key is present.

A batch is asynchronous by design, so a silent wait would be
indistinguishable from a hung test. Each poll logs the elapsed time,
the status and the counters, and past the budget the test aborts rather
than fails: a slow queue on Anthropic's side is not a regression here.
Only one test waits, so the suite waits once.

Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Add a dedicated anthropic-batch.adoc page rather than growing
anthropic-chat.adoc, which was already the largest page under api/chat
at over 1700 lines. It is nested under the Anthropic entry in the
navigation, the shape the embeddings section already uses for Bedrock
and VertexAI, and anthropic-chat.adoc keeps a short pointer so the
feature stays discoverable from the provider's entry page.

The page covers submitting a batch, polling it, streaming the results,
cancelling and deleting, the auto-configuration properties, and the two
constraints callers have to design around: correlation by customId
because results are unordered, and the absence of an interactive
tool-execution loop.

Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants